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approachabi 1 i ty ; user experience, education, and feedback; 
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paper will attempt to cover three issues: 

- A definition of benchmarking and its relation- 
ship to quality management 

- A description of a project which applied the 
technique in a library context 

- within these an exploration of the relationship 
between performance measurement and bench- 
marking 

Benchmarking Definitions and 
Quality Management Context 

T he word ‘benchmark’ originated from a survey- 
or’s mark cut to indicate a level for the determi- 
nation of altitude. In this sense a benchmark is an 
absolute measure. A distinction should be drawn 
between this usage and ‘benchmarking’ as a man- 
agement technique, in which measurement is pri- 
marily comparative. Benchmarking’s origins have 
been linked to the Japanese word ‘dantotsu’ mean- 
ing ‘striving for the best of the best’, and also to 
Sun Tzu’s Art of War and the aphorism ‘if you 
know your enemy and know yourself, you need not 
fear the result of a hundred battles’. Modem defini- 
tions of benchmarking are sometimes less poetic but 
more informative: for example Karlof and 
Ostblom’s: ‘a continuous systematic process where- 
by a comparison is made between productivity, 
quality and practices in your organisation and a cho- 
sen similar organisation’ (1993). Roger MiUiken’s 
pithy but accurate ‘stealing shamelessly’ is a sharp- 
er summary (1991). Xerox were the first company 
to begin benchmarking as a response to the crisis in 
their industry produced by the entry of the Japanese 
into the photocopier market. The Xerox benchmark- 
ing template of What? - Who is best at it? - How do 
we do it? - How do they do it? sums up the process 
the company adopted (Camp, 1989). Some interest 
has been shown within the library community and 
theoretical articles have appeared on benchmarking 
(Shaughnessy, 1993). Courses have been run in the 
UK (Library Association) and North America 
(University of Toronto Faculty of Information 
Studies), and Muir has produced a series of simple 



guides for US librarians (1993-1994). No systematic 
attempts at library benchmarking had apparently 
been made in the UK until the Royal Military 
College of Science (RMCS) Library began its pro- 
ject in late 1993. 

Four types of benchmarking were originally sug- 
gested by Camp (op. cit.): 

- ‘internal’, in which the organisation seeks uni- 
form good practice through internal compar- 
isons 

- ‘competitive’, in which specific processes are 
compared to competitors in the same industry 

- ‘fimctional’, in which specific processes are 
compared against the best at that process (irre- 
spective of industry) 

- ‘generic’, in which aU processes or functions are 
measured against the best 

Four types of benchmarking exercise have been 
identified (Zairi, 1992): 

- ‘cost driven’, which is used for ^onomic gains 

- ‘process driven’, which is motivated by desire 
for superior performance 

- ‘quick dip’, which looks for immediate bottom 
line results, and 

- ‘competitive’, in which true competitive gaps are 
sought to drive improvement in product or 
process. 

Different authorities suggest a variety of methodolo- 
gies: Zairi (and Leonard,1994) outlines processes 
for benchmarking involving from five up to 36 
steps. We opted for the 15-step approach described 
by Oakland (1993), although aU the methods con- 
tain similar elements: deciding what to benchmark; 
identifying partners; gathering information; 
analysing what has been collected; and then imple- 
menting improvements. The result is then checked 
or reviewed and the cycle may be repeated. 

Total quality management (TQM) provides a con- 
text for benchmarking. Termer and DeToro (1992) 
suggest TQM consists of three main elements: 

- Customer Focus 

- Process Improvement 

- Total Involvement 
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leading to 

- Continuous Improvement 

Customer Focus can be considered to involve the 
following activities: 

- Identifying Customers 

- Understanding Customer Expectations 

- Understanding Customers 

- Benchmarking 

Thus benchmaiking is primarily about improving 
customer focus through seeing how others satisfy 
their customers. The performance measurement 
aspects of benchmarking should be viewed as sec- 
ondary to this aim. Benchmarking can also be seen 
as a high-level proactive mechanism for improved 
customer understanding (ibid.): 

- Level 1 (Low) Most Reactive 

- Unsolicited complaints 

- Level 2 (Mid) 

- Service desks/hotlines 

- Sales data 

- Unstructured surveys 

- Level 3 (High) Most Proactive 

- Personal Interviews/Focus Groups 

- Designed Surveys 

- Benchmarking 

- ‘Mystery shopper’ 

T airi (and Leonard, 1994) suggests that for a bench- 
maiking exercise to be successful the following 
might have been achieved: 

- Impact on customer satisfaction 

- Contribute to raising competitive standards 

- Create the learning organisation 

- Inspiration from best-in-class 

- Strengthen weaker processes 

- Enhance knowledge pool 



- Bring in state-of-the-art practices 

- Keep the organisation externally focused 

- Extending employees’ creative contributions 

Cherret (1994) suggests a number of principles to 
guide a benchmarking exercise: 



- Commitment 

- Right reason . 

- Right activities 

- Understanding 



. . willingness to change 
. . desire to learn and improve 
. . specific imperatives 
. . of own processes 



- Emulate . . and improve on leaders 

- Change culture . . to targets relating to best 

- Be ethical . . honest, open exchange 

with partners 

The RMCS Library Benchmarking 
Project 

T he Library of the Royal Military College of 
Science is operated by Cranfield University for 
the UK Ministry of Defence (MoD) under an overall 
contract for the academic activity on the 
Shrivenham site (Town, 1987). As a result of the 
University's Quality Policy for its Shrivenham cam- 
pus the Library initiated a Total Quality 
Management prograimne in Summer 1993 (Town, 
1994). The characteristics of our TQM approach are 
to use the ‘road map’ taken from Termer and 
DeToro (op. cit.), and to employ a team approach 
through Quality Improvement Teams (similar to 
(Quality Circles) using a standard process improve- 
ment checklist. We also developed a ten-point 
(Quality Policy which defined quality and stressed 
the importance of staff involvement to the initiative. 
We were seeking an input of energy and creativity 
via the initiative rather than an increase in bureau- 
cracy; as a small organisation using our own inter- 
nal resources we rejected the BS 5750/ISO 9000 
approach as insufficiently focused on quality of ser- 
vice and too expensive in resources when set 
against the possible benefits. The Library had a 
strong record of excellence in customer care and of 
iimovation in service, and we intended to build on 
this by taking a more systematic and managed 
approach to continuous improvement whilst ensur- 
ing that all staff were fully involved and effectively 
led. 

Why did we engage in this major project? Our 
quality policy recognises that on the Shrivenham 
Campus we operate in an increasingly competitive 
environment. Thus the pursuit of a quality service 
recognisable by customers (students and staff) and 
demonstrable to clients (the MoD and other spon- 
sors) is essential. It is clear that the former will 
regard high quahty of service as more important, 
but that the latter will be also interested in cost con- 
trol and optimum pricing. We had some concern at 
the outset that the existing performance measures or 
indicators in use in libraries might be inadequate for 
benchmarking purposes. 

The elements of Oakland’s 15 Stage Process (op. 
cit.) are: 
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- Plan 


7 Stages 


- Analyse 


3 Stages 


- Develop 


2 Stages 


- Improve 


1 Stage 


- Review 


2 Stages 



The stages within the planning phase are: 

- Select process groups 

- Identify best competitor 

- Identify benchmarks 

- Bring together team 

- Decide information and data-coUection 
methodology 

- Prepare for visits and interact with target 

organisations 

- Use data-coUection methodology 

Stage two suggests identifying the best competitor. 
Shaughnessy points out that ‘it is not known which 
libraries, within the major types, provide the best 
service, and are therefore able to serve as bench- 
marks’ (op. cit.). Few libraries seem witling to label 
themselves as ‘best’ at any particular aspect of their 
work, even if they think it in private. Information 
from published statistics and so-called performance 
indicators do not readily identify best-in-class. For 
example the CVCP/HEFCE indicators for ‘old’ UK 
universities are based solely on expenditure 
(Committee . . ) and provide no information on cus- 
tomer satisfaction, library outputs or outcomes. 

We therefore chose to approach 60 libraries in 
various relevant groups to act as benchmarking 
partners, in the hope that we woitld be able to devel- 
op a shortlist of about 20. These included: libraries 
in technological universities; small academic 
libraries; those that had some claim to excellence, 
or relevant in the sense of being involved in quality 
initiatives or active in performance measurement; 
special libraries, including a number which had 
been recognised through awards for quality; and a 
group of academic libraries with whom we already 
had some relationship which we believed would 
make up the list should we faU short of our target 
number. 

As a result of our initial approaches 30 libraries 
agreed to participate. From this we selected a short- 
list of 20 (although in the end we were unable to 
visit all of these) on the basis of their ability to 
accommodate visits during our data collection peri- 
od (October to December 1993), their relevance to 
our situation, and also on geography. In order to 



meet our timetable we chose a group of university 
libraries in the M4 corridor, a group in the 
Midlands, and a group in the North which we could 
visit in a single ‘road-trip’ week. 

The participants were: 



Bath Aston 

Bristol De Montfort 



West of England 
South Bank 

Surrey 

Thames VaUey 



Keele 

Leicester 

Loughborough 

Warwick 



Bradford 

Central 

Lancashire 

Huddersfield 

Manchester 

Business School 

Northumbria 



Brunei, City and SOAS also completed question- 
naires but we were unable to arrange visits within 
the timescale. Other libraries were also generous in 
providing relevant information or advice in support 
of the exercise, particirlarly London University and 
the Institute of Development Studies. 

The next suggested stage is to identify ‘bench- 
marks’ or measures to use. As already mentioned 
cost and efficiency data is no longer sufficient to 
measure quality. Oakland suggests that the old 
financial measures are ‘harmful’ and ‘incompatible 
with quality improvement measures such as process 
and throughput times, delivery performance . . . and 
increases in flexibility, which are first and foremost 
non-financial’ (op. cit.). Zairi suggests that perfor- 
mance measurement ‘is not about counting, collect- 
ing absolute data, or building league tables’ (1994). 
Shaughnessy (op. cit.) concludes on the basis of 
Parasuraman, Zeithaml and Berry’s (1990) work 
that academic service quality might be defined in 
terms of the following characteristics: ‘reliability or 
consistency, timeliness, competence, access, cour- 
tesy, communication, credibility . . . and the overall 
fit between the customer’s needs and the service’ 
and that therefore ‘the most important and relevant 
data will be that provided by library users’. 

One of the standard measurement methods in 
TQM is to use a Critical Success Factors (CSFs) 
approach. We developed CSFs for the library in the 
early stages of the TQM initiative; these were inter- 
nally generated by staff but are similar to those 
developed in other academic libraries active in the 
quality arena. RMCS Library CSFs are: 



- We must provide current, accessible information 
resources which match user needs 



- We must provide cost-effective services which 
match contract requirements 

- We must have well-trained, motivated and 
approachable staff 
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- We must have effective communication with 
users 

- We must respond positively to change 

- We must provide the right environment for 
learning 

From this list it was possible to develop some possi- 
ble areas to measure for the benchmarking exercise 
in relation to each Critical Success Factor: 

- Availability of up-to-date stock 

- Unit Costs 

- Staff Development, Ability and Approachability 

- User Experience, Education and Feedback 



- Innovation 

- Learning Environment 

Because of the short timescale and the fact that this 
was also research into benchmarking as well as an 
attempt to benchmaric, the decision was taken to 
employ external consultants. We were however 
committed to involving our own staff at all levels to 
ensure that the final product was owned by the staff 
and that long-term links could be made with the tar- 
get organisations. It was also important to maintain 
the connection between the benchmaiking exercise 
and the TQM initiative. 

The Study Methodology was to gather prelimi- 
nary data through a questionnaire. The elements of 
the questionnaire could be related to the CSFs, so 
that information was sought, for example, on the 
availability of unit costs, or the degree to which a 
service was innovative. The immediate feedback 
ftorn the completed questionnaires was encourag- 
ing. It suggested first and foremost that we were 
dealing with organisations with similar concerns to 
ours. A munber of common themes emerged which 
provide a snapshot of the concerns of that particular 
period: the new approach to user surveys, the use of 
external consultants to generate change, the quest 
for realistic performance indicators, and the conver- 
gence of library and computing services. One nega- 
tive finding was the lack of availability of unit 
costs. Some participants claimed that it would be 
possible to deduce them, but no library regularly 
collected data in this particular form. An early deci- 
sion was taken therefore not to pursue this aspect 
further. The degree of honesty and openness in com- 
pleting the questionnaire, and indeed throughout the 
process, was marked. This indicated the general 
enthusiasm of the partners for using the exercise as 
an opportunity for learning and sharing experience 
and data. 



O 




The Follow-up visit was designed to elaborate the 
information provided by the questiormaire, to dis- 
cuss the general issues of quality, benchmarking and 
convergence with staff, and to conduct three sepa- 
rate measurement studies. As a result the 
Consultants, who alone attended all visits, devel- 
oped a shortlist of those libraries which they consid- 
ered to be best-in-class for a particular process. 
These were grouped around four key processes 
which were based on the CSFs and defined as: 

- User Induction and Education 

- Information Retrieval 

- Information Provision and Delivery 

- Facilities Provision 

Various constituent factors or sub-processes were 
recognised within these. Thus we created a list of 
libraries which we could use as exemplars. 

The three measurement studies were designed to 
try and quantify user-related measures so that com- 
parisons could be made across all the libraries in the 
study. This would also provide a more rigorous and 
reproducible basis for identifying best-in-class per- 
formance. 

The measurement studies were: 

- Availability Study 

- Unobtmsive Testing of enquiry services or 

‘Mystery Shopper’ 

- ‘Servqual’ type walk-through assessment 

ReviU suggested that ‘ ... to produce more mean- 
ingful comparative performance data it is strongly 
recommended that an availability study should be 
adopted as a future component of academic libraries 
performance assessment’ (1987). The rationale for 
this study was that availability and accessibility of 
materials, its currency and the speed or delays in 
delivery are critical issues for the user. The study 
was designed to give a measure of probability that a 
user would leave a library having located and 
obtained the books he or she was initially seeking. 
This required the study team to search for the books 
on the OPAC, investigate their circulation status and 
if available locate them on the shelves. 

Unobtmsive testing, or ‘mystery shopping’ to use 
modem quality jargon, was carried out in those 
libraries which agreed (16 out of 18). The method- 
ology was to prepare up to three reference enquiries 
per library and assess the degree of success in deal- 
ing with the request. The two main queries were 
specific in the sense of the tester being aware of the 
existence of certain information which would satis- 
fy his request. Some difficulties were experienced 
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because of the degree of security most academic 
libraries now employ, but in all cases the tester was 
able to sample the enquiry service and produce a 
report describing the environment of the reference 
service, what enquiry or enquiries had been made, 
and a summary of the outcome. 

All those attending a particular visit were asked 
to score the library visited on a number of qualita- 
tive factors. These were subjective impressions of 
the library which any user might have, and we 
attempted to cover some of the dimensions suggest- 
ed in Zeithaml, Parasuraman and Berry's methodol- 
ogy for assessing service quality (op. cit.). These 
were adapted to relate to the experience of library 
use and to our critical success factors. The rationale 
was to assess a realistic spread of aspects of service 
quality which could be gathered on a ‘walk- 
through’ basis and scored on a five-point scale: 

- Approachability of staff 

- Ability of staff 

- Physical appearance of stock 

- Signing and guiding 

- Library environment 

- Ease of use of OPAC 

The results were used to rank the participating 
libraries in relation to our service to provide further 
evidence of best-in-class performance. 

On completion of the exercise each partner 
library was provided with a written report sum- 
marising the findings for their library. The 
Consultants completed their task with a brief report 
at the end of 1993, and this is being incorporated 
into a final full report of the project (Town . . . ). 

Conclusions 

A t the outset we were forced to consider how we 
could claim to be seeking continuous improve- 
ment without using what industry seemed to consid- 
er to be one of the simplest but most effective tools 
available. The Consultants suggested that the pro- 
ject had been a first pass v^ch raised more ques- 
tions than it answered. With the benefit of a longer 
period to consider the outcome 1 would state much 
more positively that benchmarking is essential for 
any service which might be open to comparison. 

The RMCS Library benchmarking project has 
demonstrated that benchmarking is a technique 
which can be used by libraries and can be applied to 
any or every aspect of a service. It also suggests that 
generally libraries seem willing to act as partners. 
We are most grateful to all the participants in our 



exercise; benchmarking cannot be done without 
partners. 

DeToro (1995) identifies ten pitfalls which may 
adversely affect the success of benchmarking exer- 
cises: 

- Lack of sponsorship 

- Wrong people in the team 

- Teams not fully understanding their own work 

- Teams taking on too much 

- Managers failing to understand the necessary 
commitment 

- Focusing on metrics rather than processes 

- Benchmarking not part of larger strategy 

- Misunderstanding organisational mission, goals 
and objectives 

- Assuming every project requires a visit 

- Failure to inspect 

The project had strong organisational support from 
senior management both within and outside the 
library. Staff involvement in the teams is also criti- 
cal; using Consultants was essential to meet the 
timetable and they added a great deal of theoretical 
knowledge of both libraries and quality manage- 
ment that we would have taken a long time to accu- 
mulate. However the project was not as strongly 
owned by staff as a result and this might have inhib- 
ited both take-up of learning and individual commit- 
ment to continuing the process through follow-up 
visits. Whilst staff had clear agendas for the project, 
it would have been helpful to have been further 
along the TQM route before commencing bench- 
marking, with more specific processes defined and 
mapped. We probably need to focus our benchmark- 
ing in future at the sub-process level; that is to 
choose a functional approach. We should also 
broaden the partners to include other industries who 
use similar processes. It might also be s^d that we 
took on too much; but the project was conceived 
partly as an experiment or research on behalf of the 
library community. 

The fifth pitfall concerns performance measure- 
ment. Using currently available national perfor- 
mance indicators or statistics would almost certainly 
result in over-concentration on the metrics, because 
they do not reveal much detail of the processes 
involved. Libraries need to develop more measures 
which relate directly to the user’s experience, and to 
solve the problem of how local satisfaction mea- 
sures which are not absolute (because they are 
dependent on local expectations) can be used in 
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comparisons between different institutions. The 
development of a UK Library Benchmaiking 
Consortium might help in this respect. In addition 
libraries need to recognise the concept of the ‘capa- 
ble process’ from the user viewpoint, instead of, for 
example, allowing external suppliers dictate the 
standards of speed of document supply. 

One of the lessons learned is that benchmarking 
can be done through public sources, and that visits 
are not necessary in every case. Published statistics 
can be used for limited comparisons, but the current 
trend amongst UK academic library staff for seek- 
ing best practice is through the various electronic 
(Mailbase) discussion lists. Whilst the word ‘best’ is 
rarely explicitly used, the methodology of seeking 
partners to solve a particular problem or improve a 
particular process, collecting data from them, shar- 
ing it ethically, and providing feedback on the suc- 
cess of the chosen solution seems to me to be an 
electronic form of benchmarking. 

The benchmarking exercise did trigger a number 
of specific improvements in our service. During the 
ensuing year and a half a number of developments 
and enhancements have taken place as a result of 
the experience of others. There is still scope for 
more to be done given the huge amount of informa- 
tion collected. We certainly gained a great deal of 
confidence, reassurance and tangible evidence about 
the relative quality of our service from the exercise, 
and also a strong sense of the common issues and 
concerns within our ‘industry’. A library service 
which has benchmarked itself against its peers and 
acts on the information gained will have less con- 
cerns about its own performance and can face a 
future of increasing competition in information pro- 
vision with a greater degree of confidence. 
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